What Is Claimed Is: 



1 . A method of flow controlling InfiniBand receive traffic, 
comprising: 

maintaining a single memory structure for queuing InfiniBand traffic 
received via multiple virtual lanes and multiple queue pairs; 

identifying a first packet payload received via a first virtual lane and a first 
queue pair; 

determining whether the first payload can be stored in the memory 
structure without exceeding a portion of the memory structure allocated to the first 
virtual lane; 

determining whether the first payload can be stored in the memory 
structure without exceeding a portion of the memory structure allocated to the first 
queue pair; and 

if storing the first payload in the memory structure would exceed said 
portion of the memory structure allocated to the furst queue pair, determining 
whether the first queue pair is enabled to use a shared portion of the memory 
structure to store payloads of packets received via the first queue pair. 

2. The method of claim 1, further comprising: 

allocating a portion of the memory structure to each of the multiple virtual 
lanes; and 

allocating a portion of the memory structure to each of the multiple queue 

pairs. 

3. The method of claim 1 , wherein the memory structure comprises a 
set of linked lists of memory structure buffers, including one linked list for each of 
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the multiple queue pairs that are active. 

4. The method of claim 1 , further comprising: 

dropping the first payload if the first payload cannot be stored in the 
memory structure without exceeding the portion of the memory structure allocated 
to the first virtual lane. 

5. The method of claim 1 , further comprising: 

issuing a Retry, Not Ready, Negative Acknowledgement (RNR-NAK) if: 
the first payload cannot be stored in the memory structure without 
exceeding a portion of the memory structure allocated to the first queue 
pair; and 

the first queue pair is not enabled to use the shared portion of the 
memory structure. 

6. The method of claim 1, further comprising: 

issuing a Retry, Not Ready, Negative Acknowledgement (RNR-NAK) if: 
the first payload cannot be stored in the memory structure without 
exceeding a portion of the memory structure allocated to the first queue 
pair; 

the first queue pair is enabled to use the shared portion of the 
memory structure; and 

the shared portion of the memory structure is full. 

7. The method of claim 1 , further comprising: 

defining one or more dedicated thresholds in the portion of the memory 
structure allocated to the first queue pair; and 
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for each of said dedicated thresholds, identifying a number of message 
credits the queue pair may advertise when the amount of the memory structure 
used by the queue pair exceeds said dedicated threshold, 

8. The method of claim 1, further comprising: 

defining one or more shared thresholds in the shared portion of the 
memory structure; and 

for each of said shared thresholds, identifying a number of message credits 
the queue pair may advertise when the amount of the shared portion used by the 
multiple queue pairs exceeds said shared threshold. 

9. The method of claim 1, further comprising: 

receiving a request on a second queue pair to perform an RDMA (Remote 
Direct Memory Access) Read operation; and 

based on an amount of data expected to be received via the RDMA Read 
operation, reserving a sufficient number of buffers in the memory structure. 

10. The method of claim 1, further comprising: 

in the single memory structure, reassembling the queued InfiniBand traffic 
into outbound communications; 

receiving a payload on an idle queue pair, wherein a queue pair is idle if no 
traffic firom the queue pair is stored in the single memory structure; and 

only queuing the payload in the single memory structure if sufficient space 
in the single memory structure is reserved for completing reassembly of outboimd 
communications on each non-idle queue pair. 

11. A computer readable medium storing instructions that, when 



Attorney Docket No. SUN-P8922 



52 



Inventor: Mott 



executed by a computer, cause the computer to perform a method of flow 
controlling InfiniBand receive traffic, the method comprising: 

maintaining a single memory structure for queuing InfiniBand traffic 
received via multiple virtual lanes and multiple queue pairs; 

identifying a first packet payload received via a first virtual lane and a first 
queue pair; 

determining whether the first payload can be stored in the memory 
structure without exceeding a portion of the memory structure allocated to the first 
virtual lane; 

determining whether the first payload can be stored in the memory 
structure without exceeding a portion of the memory structure allocated to the first 
queue pair; and 

if storing the first payload in the memory structure would exceed said 
portion of the memory structure allocated to the first queue pair, determining 
whether the first queue pair is enabled to use a shared portion of the memory 
structure to store payloads of packets received via the first queue pair. 

1 2. The computer readable medium of claim 1 1 , wherein the method 
fiirther comprises: 

defining one or more dedicated thresholds in the portion of the memory 
structure allocated to the furst queue pair; and 

for each of said dedicated thresholds, identifying a number of message 
credits the queue pair may advertise when the amount of the memory structure 
used by the queue pair exceeds said dedicated threshold. 

1 3 . The computer readable medium of claim 1 1 , wherein the method 
further comprises: 
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defining one or more shared thresholds in the shared portion of the 
memory structure; and 

for each of said shared thresholds, identifying a number of message credits 
the queue pair may advertise when the amount of the shared portion used by the 
multiple queue pairs exceeds said shared threshold. 

14. The computer readable medium of claim 1 1, wherein the method 
further comprises issuing a Retry, Not Ready, Negative Acknowledgement (RNR- 
NAK) only if one of: 

(a) the first payload cannot be stored in the memory structure without 
exceeding a portion of the memory structure allocated to the first queue 
pair; and 

the first queue pair is not enabled to use the shared portion of the 
memory structure; and 

(b) the first payload cannot be stored in the memory structure without 
exceeding a portion of the memory structure allocated to the first queue 
pair; 

the first queue pair is enabled to use the shared portion of the 
memory structure; and 

the shared portion of the memory structure is full. 

15. A method of flow controlling InfiniBand traffic received via 
multiple queue pairs, the method comprising: 

maintaining a single memory structure for queuing InfiniBand traffic 
received via multiple queue pairs; 

for each of the multiple queue pairs, dedicating zero or more buffers in the 
memory structure for storing InfiniBand traffic received via said queue pair; 
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for each queue pair having a number of dedicated buffers greater than 

zero: 

identifying one or more buffer thresholds between 0 and N, 
inclusive, wherein N is the number of buffers dedicated to the queue pair; 
and 

for each of said buffer thresholds, identifying an amount of credits 
said queue pair may advertise after consuming said threshold level of 
buffers; and 

within a shared portion of the single memory structure usable by a subset 
of the multiple queue pairs: 

identifying one or more shared thresholds; and 

for each of said shared thresholds, identifying an amount of credits 

each queue pair in said subset of queue pairs may advertise after said 

shared threshold level of buffers is consumed. 

1 6. The method of claim 1 5, further comprising: 

receiving a payload of a first packet, from a first queue pair, for storage in 
the memory structure; and 

determining if the payload can be stored without exceeding said number of 
buffers dedicated to the first queue pair. 

1 7. The method of claim 1 6, further comprising: 

issuing a Retry, Not Ready, Negative Acknowledgement (RNR-NAK). 

18. The method of claim 1 6, further comprising: 

if the payload cannot be stored without exceeding said number of buffers 
dedicated to the first queue pair, determining whether the first queue pair is 
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permitted to use said shared portion of the memory structure; and 

if the first queue pair is permitted to use said shared portion of the memory 
structure, determining whether the payload can be stored without exceeding a final 
threshold in said shared portion. 

1 9. The method of claim 1 8, further comprising: 

issuing a Retry, Not Ready, Negative Acknowledgement (RNR-NAK). 

20. The method of claim 1 5, further comprising: 

receiving an instruction to perform an RDMA (Remote Direct Memory 
Access) Read operation on a first queue pair; and 

reserving a sufficient numbers of buffers in the memory structure to store 
data to be retrieved via the RDMA Read operation. 

2 1 . The method of claim 20, further comprising, before said reserving: 
calculating the sum of the number of buffers in the memory structure 

currently used by the first queue pair and the number of buffers in the memory 
structure currently reserved for other RDMA Read operations for the first queue 
pair; and 

comparing said sum to said number of buffers dedicated to the first queue 

pair. 

22. The method of claim 2 1 , further comprising: 

determining whether the first queue pair is permitted to use said shared 
portion of the memory structure. 

23. A computer readable medium storing instructions that, when 
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executed by a computer, cause the computer to perform a method of flow 
controlling InfiniBand traffic received via multiple queue pairs, the method 
comprising: 

maintaining a single memory structure for queuing InfiniBand traffic 
received via multiple queue pairs; 

for each of the multiple queue pairs, dedicating zero or more buffers in the 
memory structure for storing InfiniBand traffic received via said queue pair; 

for each queue pair having a number of dedicated buffers greater than 

zero: 

identifying one or more buffer thresholds between 0 and N, 
inclusive, wherein N is the number of buffers dedicated to the queue pair; 
and 

for each of said buffer thresholds, identifying an amount of credits 
said queue pair may advertise after consuming said threshold level of 
buffers; and 

within a shared portion of the single memory structure usable by a subset 
of the multiple queue pairs: 

identifying one or more shared thresholds; and 

for each of said shared thresholds, identifying an amount of credits 

each queue pair in said subset of queue pairs may advertise after said 

shared threshold level of buffers is consumed. 

24. The computer readable medium of claim 23, wherein the method 
further comprises: 

receiving an instruction to perform an RDMA (Remote Direct Memory 
Access) Read operation on a first queue pair; and 

reserving a sufficient numbers of buffers in the memory structure to store 
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data to be retrieved via the RDMA Read operation. 

25. The computer readable medium of claim 23, wherein the method 
further comprises: 

5 receiving a payload of a first packet, firom a first queue pair, for storage in 

the memory structure; 

determining if the payload can be stored without exceeding said number of 
buffers dedicated to the first queue pair; 

if the payload cannot be stored without exceeding said number of buffers 
1 0 dedicated to the first queue pair, determining whether the first queue pair is 
permitted to use said shared portion of the memory structure; and 

if the first queue pair is permitted to use said shared portion of the memory 
structure, determining whether the payload can be stored without exceeding a final 
threshold in said shared portion. 

15 

26. A method of flow controlling InfiniBand traffic received via 
multiple virtual lanes, comprising: 

maintaining a single memory structure for queuing InfiniBand traffic 
received via multiple active virtual lanes; 
20 for each of the multiple active virtual lanes, identifying a threshold number 

of buffers in the memory structure dedicated to storing InfiniBand traffic received 
via said virtual lane; and 

for each of the multiple active virtual lanes: 

receiving a value representing the total blocks sent on the virtual 
25 lane from the link partner of said virtual lane; 

storing said value; 

incrementing said value for each payload received via said virtual 
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lane and stored in the memory structure; 

adding to said value the number of buffers in said dedicated set of 
buffers that are unused to calculate a credit limit; and 

advertising said credit limit to the link partner. 

27. The method of claim 26, further comprising: 

for each of the multiple active virtual lanes, maintaining a used buffer 
count to track the number of memory structure buffers used to store packet 
payloads received via said virtual lane. 

28. The method of claim 27, further comprising: 
receiving a payload of a packet received on a first virtual lane; 
determining whether the payload can be stored in the memory structure 

without causing the used buffer count to exceed said threshold number of buffers. 

29. A computer readable medium storing instructions that, when 
executed by a computer, cause the computer to perform a method of flow 
controlling InfiniBand traffic received via multiple virtual lanes, the method 
comprising: 

maintaining a single memory structure for queuing InfiniBand traffic 
received via multiple active virtual lanes; 

for each of the multiple active virtual lanes, identifying a threshold number 
of buffers in the memory structure dedicated to storing InfiniBand traffic received 
via said virtual lane; and 

for each of the multiple active virtual lanes: 

receiving a value representing the total blocks sent on the virtual 

lane from the link partner of said virtual lane; 
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storing said value; 

incrementing said value for each payload received via said virtual 
lane and stored in the memory structure; 

adding to said value the number of buffers in said dedicated set of 
5 buffers that are unused to calculate a credit limit; and 

advertising said credit limit to the link partner. 

30. A method of avoiding locking, in receive InfiniBand queues, the 
method comprising: 

maintaining a single memory structure for reassembling InfiniBand traffic 
received via multiple virtual lanes and multiple queue pairs; 

identifying a first packet payload received via a furst queue pair that is idle, 
wherein the first queue pair is considered idle if no traffic from the first queue pair 
is stored in said single memory structure; 

for each other queue pair for which traffic from said queue pair is stored in 
said single memory structure, determining whether sufficient space in the single 
memory structure is reserved for reassembling said traffic; and 

storing the first packet payload in said single memory structure only if 
sufficient space in the single memory structure is available for reassembling said 
traffic. 

3 1 . The method of claim 30, wherein said determining comprises, for 
each said other queue pair: 

identifying an amount of space in said single memory structure reserved 
25 for said other queue pair; and 

comparing said amount of reserved space to an amount of space expected 
to be needed to complete reassembly of said traffic from said other queue pair. 
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32. An apparatus for flow controlling received InfiniBand traffic, 
comprising: 

a single memory structure configured to queue payloads of InfiniBand 
traffic received via multiple virtual lanes and multiple queue pairs; 

a resource manager configured to manage the memory structure; 

a first module configured to facilitate the advertisement of virtual lane 
credits; and 

a second module configured to facilitate the advertisement of queue pair 

credits. 

33. The apparatus of claim 32, wherein said single memory structure 
comprises multiple linked lists of memory structure buffers, including one linked 
list for each of the multiple queue pairs that is active. 

34. The apparatus of claim 32, wherein said first module comprises an 
InfiniBand link core. 

35. The apparatus of claim 32, wherein said second module comprises 
an acknowledgement generator configured to generate transport layer 
acknowledgements. 

36. The apparatus of claim 32, further comprising a processor interface 
configured to facilitate the programming of operating parameters associated with 
the multiple virtual lanes and the multiple queue pairs. 

37. The apparatus of claim 32, further comprising: 
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a first memory configured to store one or more parameters associated with 
operation of a first virtual lane. 

38. The apparatus of claim 37, wherein said one or more parameters 
include: 

a count of the number of memory structure buffers currently used to store 
payloads of packets received via the first virtual lane; and 

a threshold, wherein a first packet is dropped if storing the payload of the 
first packet would cause said count to exceed said threshold. 

39. The apparatus of claim 32, fiirther comprising: 

a second memory configured to store, for each of the multiple queue pairs 
that is active, one or more parameters associated with operation of said queue 
pair. 

40. The apparatus of claim 39, wherein said one or more parameters 
include: 

a maximum number of message credits advertisable by said queue pair; 

a maximum nimiber of memory structure buffers dedicated to storing 
payloads of packets received via said queue pair; and 

an indicator configured to indicate whether said queue pair is enabled to 
use a set of shared memory structure buffers. 

4 1 . The apparatus of claim 40, wherein said one or more parameters 
fiirther include: 

one or more dedicated thresholds, wherein each said dedicated threshold 
identifies a subset of said maximum nxmiber of memory structure buffers; and 
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for each said dedicated threshold, a number of message credits 
advertisable by said queue pair when said queue pair uses said subset of said 
maximum number of memory structure buffers. 

42. The apparatus of claim 40, wherein said one or more parameters 
further include: 

a number of shared memory structure buffers in said set of shared memory 
structure buffers, wherein said shared memory structure buffers are available for 
use by said queue pair to store payloads of packets received via said queue pair if: 
said queue pair has used said maximum number of memory 
structure buffers; and 

said indicator indicates that said queue pair is enabled to use said 
set of shared memory structure buffers; and 

a maximum number of message credits advertisable by said queue pair 
when said queue pair starts using said shared memory structure buffers. 

43. The apparatus of claim 42, wherein said one or more parameters 
further include: 

one or more shared thresholds, wherein each said shared threshold 
identifies a subset of said number of shared memory structure buffers; and 

for each said shared threshold, a number of message credits advertisable 
by said queue pair when said queue pair uses said subset of said number of shared 
memory structure buffers. 

44. An apparatus for flow controlling received InfiniBand traffic, 
comprising: 

a smgle memory structure comprising multiple linked lists of memory 
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structure buffers, including one linked list for each of multiple active queue pairs; 
and 

a queue pair memory configured to store, for each of the multiple active 
queue pairs, a set of parameters for facilitating use of said single memory 
structure, said parameters including: 

a maximum number of message credits issuable for the queue pair; 

a maximum number of memory structure buffers dedicated to 
storing payloads of packets received via the queue pair; 

one or more thresholds less than said maximum number of 
dedicated memory structure bxiffers; 

for each of said one or more thresholds, a number of message 
credits issuable for the queue pair when said threshold number of memory 
structure buffers are in use; and 

an indicator configured to indicate whether a set of shared memory 
structure buffers may be used after said maximum number of dedicated 
memory structure buffers are in use. 

45. The apparatus of claim 44, further comprising: 
a set of global parameters for facilitating use of said set of shared memory 
structure buffers, the global parameters including: 

a number of shared memory structure buffers in said set of shared 
memory structure buffers; 

a maximum niunber of message credits issuable by a queue pair 
using said set of shared memory structure buffers; 

one or more shared thresholds less than said number of shared 
memory structure buffers; and 

for each said shared threshold, a number of message credits 
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issuable by a queue pair using said set of shared memory structure buffers 
when the number of shared memory structure buffers in use exceeds said 
shared threshold. 

46. The apparatus of claim 44, further comprising: 

a resource manager configured to manage use of said single memory 
structure; 

a receive packet processor configured to request one or more memory 
structure buffers from said resource manager to store a payload of a newly 
received packet; and 

a post packet processor configured to request reservation of one or more 
memory structure buffers for data to be received via an RDMA (Remote Direct 
Memory Access) Read operation. 

47. The apparatus of claim 46, wherein said resource manager 
comprises said queue pair memory. 

48. The apparatus of claim 44, further comprising: 

a virtual lane memory configured to store a set of parameters for 
facilitating queuing, in said single memory structure, of payloads of packets 
received via multiple virtual lanes. 

49. A method of preventing memory locks when storing traffic from 
multiple communication streams in a single memory structure, the method 
comprising: 

for each of a set of commimication streams received at a communication 
interface, mamtaining in a single memory structure a queue for assembling 
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payloads of packets received via the communication streams into outbound 
communications; 

receiving a first packet via a first communication stream, wherein a 
payload of the packet comprises a portion of a first outbound communication to 
be assembled; 

determining a combined depth of the queues; 

determining if a first threshold amount of space in the single memory 
structure is fi-ee; and 

if less than said first threshold amount of space is free: 

determining if another portion of the first outbound communication 

is stored in the single memory structure; and 

if no other portion of the first outbound communication is stored in 

the single memory structure, rejecting the first packet. 

50. The method of claim 49, wherein said first threshold amount of 
space is approximately equal to a Maximum Transfer Unit (MTU) in effect for 
one or more communication streams in the set of communication streams. 

5 1 . The method of claim 49, wherein said first threshold amount of 
space is approximately equal to the size of the first outbound communication. 

52. The method of claim 49, fiirther comprising, if less than said 
threshold amount of space is free: 

identifying a subset of the multiple communication streams, wherein for 
each commimication stream in the subset of communication streams an outbound 
communication is being assembled in the single memory structure, including a 
second outbound communication for a second communication stream; and 



Attorney Docket No. SUN-P8922 



66 



Inventor: Mott 



after the second outbound communication is assembled, preventing 
storage of a portion of another outbound communication for the second 
communication stream until outbound communications for a threshold number of 
the subset of conununication streams are assembled. 

53. The method of claim 49, further comprising if no other portion of 
the first outbound communication is stored in the smgle memory structure: 

recording rejection of the first packet. 

54. The method of claim 49, further comprising: 
identifying communication streams for which a packet is rejected. 

55. The method of claim 54, further comprising: 

awarding priority to the identified communication streams, for storing 
packets in the single memoiy structure, when said first threshold amount of space 
becomes free. 
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